System and method for fusing information of a captured environment

ABSTRACT

A method, apparatus and computer program product for fusing information, to be performed by a device comprising a processor and a memory device, the method comprising: receiving one or more distance readings related to the environment from a Lidar device emitting light in a predetermined wavelength; receiving an image captured by a multi spectra camera, the multi spectra camera being sensitive at least to visible light and to the predetermined wavelength; identifying within the image points or areas having the predetermined wavelength; identifying one or more objects within the image; identifying correspondence between each of the light points or areas and one of the readings; associating the object with a distance, based on the reading and points or areas within the object; and outputting indication of the object and the distance associated with the at least one object.

TECHNICAL FIELD

The present disclosure relates to fusing information related to anenvironment from two sources in general, and to using the information ina car, in particular.

BACKGROUND

Most modern cars come installed with cameras, and in particular camerasinstalled on the dashboard and thus referred to as dash cams. Thesecameras are mostly used for capturing the surrounding of the car fromwithin the car. Additional cameras are sometimes used for capturingareas not easily seen by the driver, such as areas behind or to the sideof the car.

The images captured by the cameras are often stored but not used, unlessa specific need for display arises, since the captured objects are alsoseen by the driver in real-time, thus giving the views as captured bythe cameras little added value.

Fusing information may prove useful. One such usage is displaying theimage with the information to a person, such as the driver. Anotherusage relates to operating driving or driving-assisting systems, such asadvanced driver assistant systems (ADAS), whether such systems are partof an autonomous car or are used to provide safety and conveniencefeatures in regular cars.

BRIEF SUMMARY

One exemplary embodiment of the disclosed subject matter is acomputer-implemented method for fusing information, to be performed byas device comprising a processor and a memory device, the methodcomprising: receiving one or more distance readings related to theenvironment from a Lidar device emitting light in a predeterminedwavelength; receiving an image captured by a multi spectra camera, themulti spectra camera being sensitive at least to visible light and tothe predetermined wavelength; identifying within the image points orareas having the predetermined wavelength; identifying one or moreobjects within the image; identifying correspondence between one or moreof the light points or areas and one of the readings; associating theobject with a distance, based on the reading and points or areas withinthe object; and outputting indication of the object and the distanceassociated with the object. The method can further comprise designing adisplay displaying the object and the distance, wherein said outputtingcomprises displaying on a display device. The method can furthercomprise displaying the image and the distance associated with theobject, wherein the distance is displayed in the vicinity of the objecton the display device. Within the method, designing optionally comprisesdetermining an object for which a distance is to be displayed. Withinthe method, designing optionally comprises determining graphiccharacteristics for the display. The method can further comprisedetermining a command to be provided to a system, based on the objectand distance. The method is optionally implemented within a vehicle, andthe command is optionally provided to a critical system of the vehicle.Within the method, associating the object with a distance optionallycomprises associating the object with a distance selected from amultiplicity of distances received from the Lidar. Within the method,the distance is optionally selected as a minimal distance from themultiplicity of distances. Within the method, the distance is optionallyselected as an average of the multiplicity of distances.

Another exemplary embodiment of the disclosed subject matter is anapparatus for enhancing view of an environment, the apparatuscomprising: a Lidar device emitting light in a predetermined wavelength;a multi spectra camera, the multi spectra camera being sensitive atleast to visible light and to the predetermined wavelength; and aprocessor adapted to perform the steps of: receiving one or moredistances reading related to the environment from a Lidar deviceemitting light in a predetermined wavelength; receiving an imagecaptured by the multi spectra camera; identifying within the imagepoints or areas having the predetermined wavelength; identifying one ormore objects within the image; identifying correspondence between one ofthe points or areas and one of the distance readings; associating the anobject with a distance, based on the one reading and points or areaswithin the object; and outputting indication of the object and thedistance associated with the object. The apparatus can further comprisea display device for displaying the image with the distance. Within theapparatus, the processor is optionally further adapted to design adisplay of the image with the distance to be displayed on the displaydevice. Within the apparatus, the apparatus is optionally within avehicle, and the vehicle optionally further comprises a controller forreceiving the objects and the distance and determining a command to beprovided to a system within the vehicle, based on the object anddistance. Within the apparatus, the object and the distance areoptionally provided to the system over CANBUS. Within the apparatus, thecommand is optionally provided to a brakes system or to an engine of thevehicle. Within the apparatus, the Lidar device optionally emits lightin a near infra-red wavelength. Within the apparatus, the Lidar deviceoptionally comprises a single light source creating light points on theimage. Within the apparatus, the Lidar device optionally comprises twolight sources, creating light spots on the image.

Yet another exemplary embodiment of the disclosed subject matter is acomputer program product comprising a non-transitory computer readablestorage medium retaining program instructions configured to cause aprocessor to perform actions, which program instructions implement:receiving one or more distance reading related to the environment from aLidar device emitting light in a predetermined wavelength; receiving animage captured by a multi spectra camera, the multi spectra camera beingsensitive at least to visible light and to the predetermined wavelength;identifying within the image points or areas having the predeterminedwavelength; identifying one or more objects within the image;identifying correspondence between at least one of the light points orareas and the reading; associating the object with a distance, based onthe reading and points or areas within the object; and outputting theobject and the distance associated with the object.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

The present disclosed subject matter will be understood and appreciatedmore fully from the following detailed description taken in conjunctionwith the drawings in which corresponding or like numerals or charactersindicate corresponding or like components. Unless indicated otherwise,the drawings provide exemplary embodiments or aspects of the disclosureand do not limit the scope of the disclosure. In the drawings:

FIG. 1 is a schematic illustration of a car dash board with a dash cam;

FIG. 2 is a schematic illustration of a car dash board with a dash camand a Lidar, in accordance with some embodiments of the disclosure;

FIG. 3 is a schematic illustration of a view captured by the dash camand enhanced with information from a Lidar, in accordance with someembodiments of the disclosure;

FIG. 4 is a schematic illustration of a view of a beam Lidar image, inaccordance with some embodiments of the disclosure;

FIG. 5 is a schematic illustration of a view of the beam Lidar image ofFIG. 4 , with a multiplicity of segments for which data is provided, inaccordance with some embodiments of the disclosure;

FIG. 6 is a schematic illustration of a view captured by the dash camwith identified objects, in accordance with some embodiments of thedisclosure;

FIG. 7 is a schematic illustration of the view with the identifiedobjects of FIG. 6 , and with the segments of FIG. 5 , in accordance withsome embodiments of the disclosure;

FIG. 8 is a schematic block diagram of an apparatus for combining animage captured by a multi spectral camera with additional information,in accordance with some embodiments of the disclosure; and

FIG. 9 is a schematic flowchart of a method for combining an imagecaptured by a multi spectral camera with additional information, inaccordance with some embodiments of the disclosure.

DETAILED DESCRIPTION

Dash cams are commonly used and installed in many cars. Dash camscapture the environment as seen from the dash board, which is relativelyclose to the point of view of the driver, therefore, there is littleneed to represent the captured view.

Multi-spectral capturing is a technology that is gaining popularity andbecoming more and more affordable. A multi spectral camera is a camerathat captures one or more bands of visible light and at least oneinvisible wavelength band across the electromagnetic spectrum. Thewavelengths are separated by filters or other components that aresensitive to a particular wavelength. The term multi-spectral camera asused herein also includes hyper-spectral cameras, which typicallycapture visible wavelengths and a larger number, for example at leastfive, of narrow wavelength ranges.

Thus, a multi spectral camera as used in the disclosure provides imageshaving features in visible wavelengths and in at least one invisiblewavelength or wavelength range.

Common wavelengths captured by multi spectral cameras belong to thefollowing ranges:

-   -   Visible blue (450-510 nanometers);    -   Visible Green (530-590 nanometers);    -   Visible Red (640-670 nanometers);    -   Red-Edge (705-745 nanometers);    -   Near Infrared 1 (760-900 nanometers);    -   Near Infrared 2 (860-1040 nanometers);    -   Short-wave Infrared 1 (1570-1650 nanometers);    -   Short-wave Infrared 2 (2080-2350 nanometers); and    -   Thermal Infrared (10600-12510 nanometers);

Another known technology relates to a Light Detection and Ranging(Lidar) device for remote sensing of distances, that uses pulsed laserto measure distances from the Lidar to objects, locations or surfaces inone or more directions. The Lidar technology comprises illuminating inone or more directions with light emitted by one or more light sources,and using the reflected light for determining the distance between thelight source and a target present in the illuminated direction,according to the time difference between the pulse leaving the Lidardevice and the return pulse received by the sensor. By projecting one ormore narrow laser beams in a multiplicity of directions, for example bythe beam emitter being rotated around an axis, physical features in theenvironment may be mapped with high resolutions. In some applicationsthe distances to the nearest object in a multiplicity of directionsaround the light source may be determined, thus providing a mapping ofthe light source environment.

A Lidar device can illuminate the environment in a sphere or partthereof around the device. In other embodiments, a Lidar can illuminatethe environment in a circle or part thereof, for example in parallel tothe ground.

In further embodiments, a Lidar may comprise two light sources locatedon a horizontal segment, and may combine a multiplicity, for example 16active sensing elements that together form a sensor. The returnedresults may include a value indicating a distance of objects in each ofthe 16 segments.

The laser source of the device is sometimes a green light having awavelength such as about 532 nm, or near-infrared light havingwavelength 940 nm, 1064 nm, or others.

A Lidar further comprises a sensor for receiving the light, and acomputing platform that analyses, collects or stores the data. Analysismay include converting the time-intervals to distances. Some specificLidars, such as Lidars mounted upon aircrafts may comprise additionalcomponents, such as GPS for determining the aircraft's position in termsof latitude, longitude and altitude (x, y, z coordinates), and anIntertial Measurement Unit (IMU) for determining the roll, pitch and yaw(the vertical and horizontal movements) of the aircraft in flight,wherein these measurements are required for correct computation of thedistances due to the high speed of the aircraft.

One technical problem dealt with by the disclosed subject matter is theneed to provide a driver or another user or system with exactinformation of the environment, whether the driver can see theenvironment or not. Of particular importance is the distance between thecar and objects in its vicinity. Objects may relate to static objectssuch as traffic signs, buildings, trees or the like, and also to movingobjects such as other cars, humans, animals, or the like. It will beappreciated that most objects to be considered are at the ground levelor small elevation above ground level, for example up to 50 cm.

Referring now to FIG. 1 , showing a car dash board 100, and a dash cam104 installed on the windshield and capturing the view substantially asseen by the driver. The captured images, whether still or video may bestored and played a required.

One technical solution relates to a device comprising a Lidar, amulti-spectral camera, a processing unit and optionally a displaydevice. The multi spectral unit is adapted to capture at least someparts of the visible spectrum, as well as the wavelength of the Lidar.The Lidar is placed in the vicinity of the camera, and emits light alonga horizontal plane in a range substantially equal to the field of viewof the camera, at a level such as between about 10 cm above ground and 1meter above the ground. Thus, since the camera is adapted to capture theLidar wavelength, then in addition to capturing the environment, thecamera also captures the light emitted by the Lidar and hitting objectsin the environment. The lighted points or areas can then be detected byimage analysis, and the object present at the location of the lightpoint can be associated and fused with the distance reported by theLidar. Matching the light point/area with the distance report can bedone using the time stamp associated with each such report, which shouldmatch the time stamp of a captured frame, or otherwise synchronizing theLidar and the camera.

The captured image can then be displayed to a user, with an indicationof the distance of each identified object, thus utilizing the fusedinformation.

Additionally or alternatively, the fused information can be used fordetermining commands to be provided to one or more systems of thevehicle, and in particular driving assisting systems, such as the brakesif an object is at a short distance from the car, the engine, or thelike.

Referring now to FIG. 2 , showing an illustration of a car dash boardwith a multi-spectral dash cam and a Lidar. Dashboard 100 has installedthereon a multi-spectral camera 204, and a Lidar device 208.Multi-spectral camera 204 captures the environment and the light emittedby Lidar 208 as it hits objects. The captured images are analyzed, andobjects appearing in the images are being associated with distances ascomputed by the Lidar. The images, together with the distanceindications can be displayed on any display device, such as display 212,a dedicated display installed on the windshield or the dashboard, or thelike.

It will be appreciated that the camera and the Lidar may be installedanywhere in the car and are not limited to being installed on thewindshield. Further, a multiplicity of systems such as multiplicity ofLidar and camera sets may be installed at different areas of the car.Thus the apparatus can be used in conjunction with a reverse camera,wherein the captured images together with the distances are displayedwhenever the driver drives backwards.

FIG. 3 shows a schematic illustration of a view 300 as displayed on adisplay device, comprising images as captured by a multi spectralcamera, with the distance indications 304, 306, 308, 320 associated withviewed objects provided by a Lidar having a single light source andscanning horizontally. It will be appreciated that the thick lineindicating where the Lidar light hits the objects may be invisible andis thus not displayed to a user. The line is shown in FIG. 3 forexplanatory purposes.

A technical effect of the disclosure is the enhancement of images ascaptured by a camera installed on the dash board or the windshield of acar, with useful information, and in particular the distance between thecar and various static or dynamic object in the vicinity of the car.

Another effect of the disclosure is the usage of fused information indetermining commands for operating the vehicle, such as by transmittinga command to a critical system. For example, if an object is identifiedat a distance shorter than a first threshold from the car and the car isadvancing in a velocity exceeding a second threshold, a braking commandmay be generated and sent over a communication channel such as a CANBUSto the relevant system, being the brakes. In another situation thecontroller may determine and transmit a command to the engine toaccelerate, or the like.

FIG. 4 shows a schematic illustration of an image 400 captured by amulti spectral camera capturing the light of a 2 light source Lidarhitting a dark wall, including light spot 408.

FIG. 5 shows the same image, in which the sequence of segments, such as16 segments, for which distances are provided by the Lidar, such assegments 512 and 516 are indicated on light spot 408. It will beappreciated that a distance indication is provided for each suchsegment.

FIG. 6 shows an illustration of a captured image 600. The image had beenprocessed by one or more image processing techniques, for recognizingobjects within the image, such as persons 604, 608, 612 and 616, cars620, 624 an d628, or the like.

The objects may be recognized using algorithms such as but not limitedto edge detection, object segmentation, histogram of gradient, neuralnetworks or any machine learning or deep learning algorithms.

FIG. 7 shows image 600 with the same identified objects, and also withthe segments associated with the distances, as shown on FIG. 5 . Thus,the distance to an object appearing in the image can be determined inaccordance with the distance reported by the Lidar for the one or moresegments overlapping with the object. It will be appreciated that if theobject partly overlaps with two or more segments, multiple distances maybe associated with different areas of the object. In other embodiments,a combined distance, such as an average, may be calculated for theobject. The average can be a weighted average taking into account thearea of the object within the image that overlaps with each segment. Inother embodiments, the distance associated with the object may be theminimal reported distance for any of the segments overlapping with theobject. This scheme is useful particularly in applications related todriving, in which the nearest area of an object is relevant for a driverapproaching the object.

Referring now to FIG. 8 , showing a schematic block diagram of aninformation fusing apparatus for a vehicle, in accordance with someembodiments of the disclosure.

It will be appreciated that in addition to the components detailedbelow, a vehicle in which the apparatus is installed may comprise alsothe following components that can interface or use the apparatus.

Thus, the vehicle can comprise ADAS for carrying out activities relatedto operating the car, such as braking or steering. ADAS 805 can beresponsible for all the driving operations, for example in an autonomouscar, or to providing certain features, such as cruise control or others,

The vehicle can comprise a communication channel through whichcomponents can communicate and send data or commands to each other, suchas controller area network operating with the CANBUS protocol. It willbe appreciated that other communication channels can be used, forexample Ethernet. One or more information fusing apparatuses inaccordance with the disclosure can also be connected to thecommunication channel.

The apparatus may comprise Lidar 801 and multi spectral camera 802,configure to capture at least substantial parts of the visible light, aswell as the wavelength emitted by Lidar 801. Lidar 801 may emitnear-infrared light having wavelength such as 940 nm, 1064 nm or others.

The apparatus also comprises computing device 803, which may compriseone or more processors 804. Any of processors 804 may be a CentralProcessing Unit (CPU), a microprocessor, an electronic circuit, anIntegrated Circuit (IC) or the like. Alternatively, computing device 803can be implemented as firmware written for or ported to a specificprocessor such as digital signal processor (DSP) or microcontrollers, orcan be implemented as hardware or configurable hardware such as fieldprogrammable gate array (FPGA) or application specific integratedcircuit (ASIC). Processors 804 may be utilized to perform computationsrequired by apparatus 400 or any of it subcomponents.

Computing device 803 may comprise one or more storage devices 812 forstoring executable components, and which may also contain data duringexecution of one or more components. Storage device 812 may bepersistent or volatile. For example, storage device 812 can be a Flashdisk, a Random Access Memory (RAM), a memory chip, an optical storagedevice such as a CD, a DVD, or a laser disk; a magnetic storage devicesuch as a tape, a hard disk, storage area network (SAN), a networkattached storage (NAS), or others; a semiconductor storage device suchas Flash device, memory stick, or the like. In some exemplaryembodiments, storage device 812 may retain data structures and programcode operative to cause any of processors 804 to perform acts associatedwith any of the steps shown in FIG. 9 below.

The components detailed below may be implemented as one or more sets ofinterrelated computer instructions, executed for example by any ofprocessors 804 or by another processor. The components may be arrangedas one or more executable files, dynamic libraries, static libraries,methods, functions, services, or the like, programmed in any programminglanguage and under any computing environment.

In some exemplary embodiments of the disclosed subject matter, storagedevice 812 may comprise light identification module 816 for receivingone or more images captured by multi spectral camera 802, and foridentifying therein one or more points or spots associated with thewavelength emitted by Lidar 801.

Storage device 812 can comprise image analysis engines 820 for analyzingthe image and performing various algorithms, such as but not limited toedge detection, object detection histogram of gradient, neural networksor any machine learning or deep learning algorithms for identifyingobjects within the image.

Storage device 812 can comprise Lidar-image correspondence module 824,for associating one or more objects with corresponding one or more Lidarreadings, which may be implemented in accordance with the type of Lidarused.

For example, if a single light source Lidar is used, one or more pointsof light may be detected in an image, and the distance reported for thepoints may be associated with the object detected in the area of eachpoint.

The correspondence between a point in an image and the readingindicating its distance may be made by synchronizing the camera and theLidar.

In another embodiment, in which a two light source Lidar is used, alight spot such as shown in FIG. 5 is produced, and a distance isprovided for each of a predetermined number of segments. In thisembodiment, some of the objects in the image may be associated with oneor more readings related to segments overlapping with the objects.

Storage device 812 can comprise object distance determination module 828for associating a representative distance with an object, based on theobtained one or more readings for the object. If multiple readings areprovided for multiple points associated with the object, such as whenusing a single light source Lidar, the distance of the object may becalculated by combining the distances, for example by averaging. If aLidar with multiple light sources is used, the distance to an objectappearing in the image can be determined in accordance with the distancereported by the Lidar for the one or more segments overlapping with theobject. For example, a combined distance, such as an average may becalculated for the object. The average can be a weighted average takinginto account the area of the object within the image that overlaps witheach segment. In other embodiments, the distance associated with theobject may be the minimal reported distance for any of the points or thesegments overlapping with the object.

Storage device 812 can comprise display arrangement module 832, forarranging the display of the image and the distances, if images fusedwith additional information are to be displayed. It will be appreciatedthat although the light point or spots emitted by the Lidar are capturedby the camera, they may or may not be displayed in the image. In orderto display them, the light points, which may be invisible to humans, maybe assigned a specific color in the displayed image.

Display arrangement module 832 may be operative in determining whether adistance to an object need be displayed: for example, a faraway buildingmay be irrelevant for a driver, and displaying its distance may onlyclutter the display. For other objects, it may be determined where on ornear the object its distance is to be displayed, such that it is seen asclearly as possible, and correctly represents a distance to the object.For example, if the image shows a very large building, it may bematerial which part of the building the distance relates to.

Display arrangement module 832 may also be responsible for determining,the refresh rate of the image or the distance indications, such that itdoes not exceed a predetermined threshold, in order for a user, and inparticular a driver be able to easily grasp the number, which isimpossible if changes are too frequent. Since typical scanning speed bythe Lidar can be in the order of magnitude of 200 rounds per minute, thechanges in the distances of objects between consecutive rounds of theLidar is not significant, thus it may not be necessary to update thedistance during each round of the Lidar.

Display arrangement module 832 can also be operative in displayingdistances not only to the object for which distances are available inthe last captured image, but also to other objects in the field of view,for which the distance has been determined in a previous frames. Thismay be particularly useful with a single source Lidar, in which eachimage may show only light points in certain areas of the image

It will be appreciated that different distances may be displayed withdifferent colors or other characteristics. For example, distanced undera predetermined threshold such as 10 meters can be colored red whilefarther objects can be colored green. Moreover, if an object within theimage is closer than a predetermined threshold, a vocal indication mayalso be provided.

In some embodiments, apparatus 800 may comprise a display device 836 fordisplaying the image and the distances to a driver. Display device 432can be a dedicated display device or any other display device availablewithin the car.

Referring now to FIG. 9 , showing a flowchart of a method for displayingcaptured view with additional information, in accordance with someembodiments of the disclosure.

On step 900, system setup and calibration may be performed. Setup mayinclude synchronizing Lidar 801 and camera 802. Setup may also includecalibrating the Lidar and the camera, such that the peak reflectionamplitude of the Lidar is in the center of the image or in another knownpart of the image, mapping the segments for which distances are providedby the Lidar to areas of the frame, and calibrating the wavelengthscaptured by the camera such that the wavelength of the Lidar is capturedor the like.

On step 904, one or more images captured by multi spectral camera 802may be received.

On step 906, which may be performed before, after or in parallel to step904, one or more readings related to one or more directions or areas arereceived from Lidar 801.

On step 908, one or more points, spots or areas may be identified bylight identification module 816 within the received image, which pointshave a pixel value associated with the wavelength of the Lidar. This isparticularly useful using a single source Lidar. In embodiments in whicha light spot is created the area associated with the peak reflectionamplitude of the Lidar is identified.

On step 912, one or more objects such as humans, pets, cars, buildingsor others may be identified within a captured image, for example byobject manipulation module 820.

On step 916, correspondence may be identified by Lidar-imagecorrespondence module 824 between Lidar points identified within theimage on step 908 and objects identified on step 912. The correspondencemay be performed using time synchronization, and the calibrationperformed on setup step 900.

On step 920, one or more distances as provided by Lidar 801 for eachpoint or segment may be associated with one or more objects, inaccordance with the correspondence determined on step 916. Theassociation can be performed, for example, by object distancedetermination module 828.

On step 924 the results, comprising fusion of one or more objects anddistances to the objects may be output. In some embodiments the resultsmay be displayed over a display device. In such embodiments, a visualdisplay may be designed, comprising distances to which objects are to bedisplayed, where on a displayed image distances are to be displayed,with what color or other graphic characteristics, at what refresh rate,or the like.

Additionally or alternatively, the results may be provided, for examplevia CANBUS to a controller which may use the results to generate one ormore commands for operating the vehicle. For example, if an object suchas a human, a building or a stop sign is present at distance shorterthan a first threshold from the car and the car is advancing in avelocity exceeding a second threshold, a braking command may begenerated and sent over the communication channel such as a CANBUS tothe relevant system, being the brakes. In another situation thecontroller may determine and transmit a command to the engine toaccelerate, to an alert generation system, or the like. It will beappreciated that if a system in accordance with the disclosure is used Iother environments, such as security, commands to other units can betransmitted, For example, in a system protecting an area, an alert naybe sent when a person is approaching the fence, including the distanceof the person from the fence.

Generally, closer objects can be given higher priority such thatdistances to such objects are indicated while distances to fartherobjects can be omitted. However, other prioritization can be made. Forexample certain types of objects, such as humans or stop signs can bepreferred and distances to such objects may be indicated even if closerobjects are identified.

An apparatus in accordance with the disclosure may be combined withadditional sensors or additional processing systems, such as but notlimited to microphones and audio analysis systems, or others. The datacollected from multiplicity of sensors and processed by multiplicity ofsystems may be fused in order to further enhance data provided to a useror to controllers adapted for determining commands and sending thecommands to driving assisting systems.

It will be appreciated that the disclosed method and apparatus are notlimited to being used in cars, rather they may be used in otherenvironments such as airplanes, Internet of Things (IoT) devices,infrastructure facilities, homeland security or other security systems,military use, or any other application in which identification anddistance can be combined.

The present invention may be a system, a method, and/or a computerprogram product. The computer program product may include a computerreadable storage medium (or media) having computer readable programinstructions thereon for causing a processor to carry out aspects of thepresent invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Smalltalk, C++ or the like, andconventional procedural programming languages, such as the “C”programming language or similar programming languages. The computerreadable program instructions may execute entirely on the user'scomputer, partly on the user's computer, as a stand-alone softwarepackage, partly on the user's computer and partly on a remote computeror entirely on the remote computer or server. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the block may occur out of theorder noted in the figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present invention has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the invention. Theembodiment was chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

What is claimed is:
 1. A method for fusing information, to be performedby a device comprising a processor and a memory device, the methodcomprising: receiving at least one distance reading related to theenvironment from a Lidar device emitting light in a predeterminedwavelength; receiving an image captured by a multi spectra camera, themulti spectra camera being sensitive at least to visible light and tothe predetermined wavelength; identifying within the image points orareas having the predetermined wavelength; identifying at least oneobject within the image; identifying correspondence between at least oneof the points or areas and the at least one reading; associating the atleast one object with a distance, based on the at least one reading andpoints or areas within the at least one object, wherein associating theat least one object with a distance comprises associating the at leastone object with a distance selected from a multiplicity of distancesreceived from the Lidar, and wherein the distance is selected as anaverage of the multiplicity of distances; and outputting indication ofthe at least one object and the distance associated with the at leastone object.
 2. The method of claim 1, further comprising designing adisplay displaying the at least object and the distance, wherein saidoutputting comprises displaying on a display device.
 3. The method ofclaim 2, further comprising displaying the image and the distanceassociated with the at least one object, wherein the distance isdisplayed in the vicinity of the object on the display device.
 4. Themethod of claim 2, wherein designing comprises determining objects ofthe at least one object for which a distance is to be displayed.
 5. Themethod of claim 2, wherein designing comprises determining graphiccharacteristics for the display.
 6. The method of claim 1, wherein themethod further comprising determining a command to be provided to asystem, based on the at least one object and distance.
 7. The method ofclaim 6, wherein the method is implemented within a vehicle and whereinthe command is provided to a critical system of the vehicle.
 8. Themethod of claim 1, wherein the distance is selected as a minimaldistance from the multiplicity of distances.
 9. An apparatus for fusinginformation of an environment, the apparatus comprising: a Lidar deviceemitting light in a predetermined wavelength; a multi spectra camera,the multi spectra camera being sensitive at least to visible light andto the predetermined wavelength; and a processor adapted to perform thesteps of: receiving at least one distance reading related to theenvironment from the Lidar device; receiving an image captured by themulti spectra camera; identifying within the image points or areashaving the predetermined wavelength; identifying at least one objectwithin the image; identifying correspondence between at least one of thepoints or areas and the at least one distance reading; associating theat least one object with a distance, based on the at least one readingand points or areas within the at least one object, wherein associatingthe at least one object with a distance comprises associating the atleast one object with a distance selected from a multiplicity ofdistances received from the Lidar, and wherein the distance is selectedas an average of the multiplicity of distances; and outputtingindication of the at least one object and the distance associated withthe at least one object.
 10. The apparatus of claim 9, furthercomprising a display device for displaying the at least one image withthe distance.
 11. The apparatus of claim 10, wherein the processor isfurther adapted to design a display of the at least one image with thedistance to be displayed on the display device.
 12. The apparatus ofclaim 9, wherein the apparatus is within a vehicle, and the vehiclefurther comprises a controller for receiving the at least one object andthe distance and determining a command to be provided to a system withinthe vehicle, based on the at least one object and distance.
 13. Theapparatus of claim 12, wherein the at least one object and the distanceare provided to the system over CANBUS.
 14. The apparatus of claim 12,wherein the command is provided to a critical system of the vehicle. 15.The apparatus of claim 9, wherein the Lidar device emits light in a nearinfrared wavelength.
 16. The apparatus of claim 9, wherein the Lidardevice comprises a single light source creating light points on theimage.
 17. The apparatus of claim 9, wherein the Lidar device comprisestwo light sources, creating light spots on the image.
 18. A computerprogram product comprising a non-transitory computer readable storagemedium retaining program instructions configured to cause a processor toperform actions, which program instructions implement: receiving atleast one distance reading related to the environment from a Lidardevice emitting light in a predetermined wavelength; receiving an imagecaptured by a multi spectra camera, the multi spectra camera beingsensitive at least to visible light and to the predetermined wavelength;identifying within the image points or areas having the predeterminedwavelength; identifying at least one object within the image;identifying correspondence between each of the points or areas and theat least one reading; associating the at least one object with adistance, based on the at least one reading and points or areas withinthe at least one object, wherein associating the at least one objectwith a distance comprises associating the at least one object with adistance selected from a multiplicity of distances received from theLidar, and wherein the distance is selected as an average of themultiplicity of distances; and outputting the at least one object andthe distance associated with the at least one object.